Using a design matrix \(X\) makes writing out the math easier. \(Y\) is (usually) a \(1\times n\) vector of responses, \(X\) is a \(n \times (p+1)\) matrix of predictors…plus a little extra.
\[
y = X \beta
\] It’s just matrix shorthand for the traditional linear regression formula.
\[
y = \beta_0 + \beta_1x_1 + ...\beta_nx_n
\]
Design Matrix
\[
y = \beta_0 + \beta_1x_1 + ...\beta_nx_n
\]
🔮 notice in the formula: does changing any of the \(x_n\)s impact the other \(x\)s?
Design Matrix
\[
y = X \beta
\]
The design matrix also allows us to write this problem in a format that makes it clear that least squares can help us choose \(\beta\) in a way that minimized \(Y - \beta X\)!
Least Squares
\[
y = X \beta
\]
Often, there is no \(\beta\) that can perfectly predict \(y\) (and if there is…maybe it’s overfitting…)
Least Squares
We want to choose a line in the column space of \(X\) that is as close as possible to \(y\) as possible. In other words: minimize \(\parallel y- X\beta \parallel^2\).
Least Squares
The best \(\beta\) we can choose that is both:
in the column space of \(X\), and
minimizes the distance between \(y\) and \(X\beta\) (the error)
is the projection (“shadow”) of \(y\) on \(C(X)\).
Least Squares
So our best fit is the projection of \(y\) onto the column space \(C(X)\):
\[
X \beta = proj_{C(X)} y
\]
Least Squares
A little mathy-math
\[
X\beta - y = proj_{C(X)} y - y
\]
Least Squares
\(proj_{C(X)} y - y\) is orthogonal to \(C(X)\), and thus is in the nullspace of \(X^T\).
doing a little LA: \[
X^TX\beta - X^Ty = 0 \\
X^TX\beta = X^Ty \\
(X^TX)^{-1}X^TX\beta = (X^TX)^{-1}X^Ty \\
\]
Least Squares
visual intuition for minimizing errors
why multicollinearity is a problem
(for example, if there is perfect collinearity: total sales, US sales, non-US sales, \(X^TX\) is not invertible, \(X\) is not of full rank, and there exists multiple solutions!)
Determinants
the determinant of \(A\) is the area of the unit square after multiplying by \(A\)
\(U\) an orthogonal matrix; eigenvectors of \(AA^T\)
\(D\) is a diagonal matrix; diagonal is root of positive eigenvalues of \(AA^T\) or \(A^TA\)
\(V\) an orthogonal matrix; eigenvectors of \(A^TA\)
Singular Value Decomposition
We can use SVD to do PCA by decomposing the data matrix \(X\) rather than eigendecomposing the covariance matrix \(C = \frac{X^TX}{n-1}\) into \(VLV^T\) where \(V\) are the eigenvectors (PCs) and \(L\) contains the eigenvalues.
if \(X = UDV^T\), \(V\) contains the eigenvectors needed to create PCs, and \(D\) contains the (scaled, root) eigenvalues for each PC.
\[
C = \frac{VDU^TUDV^T}{n-1} = \\
V\frac{D^2}{n-1}V^T
\]
